Sequential Subset Matching for Dataset Distillation

Dataset distillation is a newly emerging task that synthesizes a small-size dataset used in training deep neural networks (DNNs) for reducing data storage and model training costs. The synthetic datasets are expected to capture the essence of the knowledge contained in real-world datasets such that the former yields a similar performance as the latter. Recent advancements in distillation methods have produced notable improvements in generating synthetic datasets. However, current state-of-the-art methods treat the entire synthetic dataset as a unified entity and optimize each synthetic instance equally. This static optimization approach may lead to performance degradation in dataset distillation. Specifically, we argue that static optimization can give rise to a coupling issue within the synthetic data, particularly when a larger amount of synthetic data is being optimized. This coupling issue, in turn, leads to the failure of the distilled dataset to extract the high-level features learned by the deep neural network (DNN) in the latter epochs. In this study, we propose a new dataset distillation strategy called Sequential Subset Matching (SeqMatch), which tackles this problem by adaptively optimizing the synthetic data to encourage sequential acquisition of knowledge during dataset distillation. Our analysis indicates that SeqMatch effectively addresses the coupling issue by sequentially generating the synthetic instances, thereby enhancing its performance significantly. Our proposed SeqMatch outperforms state-of-the-art methods in various datasets, including SVNH, CIFAR-10, CIFAR-100, and Tiny ImageNet. Our code is available at https://github.com/shqii1j/seqmatch

Magnetic control of the valley degree of freedom of massive Dirac fermions with application to transition metal dichalcogenides

We study the valley-dependent magnetic and transport properties of massive Dirac fermions in multivalley systems such as the transition metal dichalcogenides. The asymmetry of the zeroth Landau level between valleys and the enhanced magnetic susceptibility can be attributed to the different orbital magnetic moment tied with each valley. This allows the valley polarization to be controlled by tuning the external magnetic field and the doping level. As a result of this magnetic field induced valley polarization, there exists an extra contribution to the ordinary Hall effect. All these effects can be captured by a low energy effective theory with a valley-orbit coupling term.Comment: 9 pages, 6 figure

FreeCOS: Self-Supervised Learning from Fractals and Unlabeled Images for Curvilinear Object Segmentation

Curvilinear object segmentation is critical for many applications. However, manually annotating curvilinear objects is very time-consuming and error-prone, yielding insufficiently available annotated datasets for existing supervised methods and domain adaptation methods. This paper proposes a self-supervised curvilinear object segmentation method that learns robust and distinctive features from fractals and unlabeled images (FreeCOS). The key contributions include a novel Fractal-FDA synthesis (FFS) module and a geometric information alignment (GIA) approach. FFS generates curvilinear structures based on the parametric Fractal L-system and integrates the generated structures into unlabeled images to obtain synthetic training images via Fourier Domain Adaptation. GIA reduces the intensity differences between the synthetic and unlabeled images by comparing the intensity order of a given pixel to the values of its nearby neighbors. Such image alignment can explicitly remove the dependency on absolute intensity values and enhance the inherent geometric characteristics which are common in both synthetic and real images. In addition, GIA aligns features of synthetic and real images via the prediction space adaptation loss (PSAL) and the curvilinear mask contrastive loss (CMCL). Extensive experimental results on four public datasets, i.e., XCAD, DRIVE, STARE and CrackTree demonstrate that our method outperforms the state-of-the-art unsupervised methods, self-supervised methods and traditional methods by a large margin. The source code of this work is available at https://github.com/TY-Shi/FreeCOS.Comment: Accepted by ICCV 202

Learning Green's functions associated with time-dependent partial differential equations

Neural operators are a popular technique in scientific machine learning to learn a mathematical model of the behavior of unknown physical systems from data. Neural operators are especially useful to learn solution operators associated with partial differential equations (PDEs) from pairs of forcing functions and solutions when numerical solvers are not available or the underlying physics is poorly understood. In this work, we attempt to provide theoretical foundations to understand the amount of training data needed to learn time-dependent PDEs. Given input-output pairs from a parabolic PDE in any spatial dimension $n\geq 1$, we derive the first theoretically rigorous scheme for learning the associated solution operator, which takes the form of a convolution with a Green's function $G$. Until now, rigorously learning Green's functions associated with time-dependent PDEs has been a major challenge in the field of scientific machine learning because $G$ may not be square-integrable when $n>1$, and time-dependent PDEs have transient dynamics. By combining the hierarchical low-rank structure of $G$ together with randomized numerical linear algebra, we construct an approximant to $G$ that achieves a relative error of $\smash{\mathcal{O}(\Gamma_\epsilon^{-1/2}\epsilon)}$ in the $L^1$-norm with high probability by using at most $\smash{\mathcal{O}(\epsilon^{-\frac{n+2}{2}}\log(1/\epsilon))}$ input-output training pairs, where $\Gamma_\epsilon$ is a measure of the quality of the training dataset for learning $G$, and $\epsilon>0$ is sufficiently small.Comment: 34 pages, 3 figure

Continual Task Allocation in Meta-Policy Network via Sparse Prompting

How to train a generalizable meta-policy by continually learning a sequence of tasks? It is a natural human skill yet challenging to achieve by current reinforcement learning: the agent is expected to quickly adapt to new tasks (plasticity) meanwhile retaining the common knowledge from previous tasks (stability). We address it by "Continual Task Allocation via Sparse Prompting (CoTASP)", which learns over-complete dictionaries to produce sparse masks as prompts extracting a sub-network for each task from a meta-policy network. By optimizing the sub-network and prompts alternatively, CoTASP updates the meta-policy via training a task-specific policy. The dictionary is then updated to align the optimized prompts with tasks' embedding, thereby capturing their semantic correlations. Hence, relevant tasks share more neurons in the meta-policy network via similar prompts while cross-task interference causing forgetting is effectively restrained. Given a trained meta-policy with updated dictionaries, new task adaptation reduces to highly efficient sparse prompting and sub-network finetuning. In experiments, CoTASP achieves a promising plasticity-stability trade-off without storing or replaying any past tasks' experiences and outperforms existing continual and multi-task RL methods on all seen tasks, forgetting reduction, and generalization to unseen tasks.Comment: Accepted by ICML 202

Magnetoelectric Coupling and Electric Control of Magnetization in Ferromagnet-Ferroelectric-Metal Superlattices

Ferromagnet-ferroelectric-metal superlattices are proposed to realize the large room-temperature magnetoelectric effect. Spin dependent electron screening is the fundamental mechanism at the microscopic level. We also predict an electric control of magnetization in this structure. The naturally broken inversion symmetry in our tri-component structure introduces a magnetoelectric coupling energy of $P M^2$. Such a magnetoelectric coupling effect is general in ferromagnet-ferroelectric heterostructures, independent of particular chemical or physical bonding, and will play an important role in the field of multiferroics.Comment: 5 pages including 3 figures and 1 tabl

Performance of an omnidirectional piezoelectric wind energy harvester

This paper presents a vortex-induced vibration (VIV)-based piezoelectric energy harvester that performs well for all wind directions, a so-called omnidirectional wind energy harvester. The kinetic energy of this harvester stems from wind-induced vibrations of a circular cylinder mounted on an orthogonal bibeam system, rather than a traditional single beam. Wind tunnel testing results show that compared to the traditional single-beam energy harvester, the proposed harvester substantially enhances the effectiveness, in most cases that the beam is skew to the incoming flow. The reasons for the enhancement are explained in detail by examining the wind-induced displacement response components of the cylinder identified by the image processing technique. For all wind directions, both the maximal output energy and the range of effectively working wind speed of the proposed bibeam wind energy harvester are significantly improved with respect to the single-beam system, indicating excellent performance of the proposed omnidirectional harvester in a natural wind environment